Imaging system, server device, control method for server device, and storage medium

ABSTRACT

An imaging system including an imaging device 501 and a recording server 502 communicatively connected to the imaging device 501, wherein the imaging device 501 includes an imaging unit 503 that generates a video with a plurality of resolution, a dividing unit 504 that performs a division process of dividing the video generated by the imaging unit 503 into one or a plurality of tile areas and generates a tile image, and a transmission unit 506 that transmits the video to the recording server 502, wherein the recording server 502 includes a division control unit 507 that outputs an instruction to change a division method for the division process to the imaging device according to a designation frequency of an area designated on the video transmitted from the imaging device 501.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an imaging system, a server device, acontrol method for a server device, and a storage medium.

Description of the Related Art

Occasions in which users view videos provided from imaging devices suchas cameras remotely have increased due to the progress of networktechnologies represented by the Internet. A service that allows a userto freely view his or her observation area by letting the user designatean observation area in an image of a camera capturing a certain area tocause the client side to digitally cut out the observation area and toenlarge/reduce the area, or the like to display the area is known.

However, if the client side cuts out an area from the video, andenlarges it for display, the effective resolution of the videodecreases, and thus the video looks degraded. In order to overcome thisproblem, a group of tile videos obtained by dividing an entire video ora plurality of videos with a high resolution into a plurality ofpredetermined areas (tile areas) may be transmitted to a recordingserver. In addition, a technique of managing high scalability and abandwidth by transmitting a tile video that is closest to an observationarea designated by a user to a client is known.

According to JP 2016-58994A and JP 2018-156474A, it is possible todistribute a video corresponding to a range selected by a user byswitching between videos to be distributed according to selection or abehavior of the user.

However, if a user wants to view a video with high resolution in a casewhere there is a significant difference between each tile group capturedusing an imaging device such as a camera and an area selected(designated) by a user, it is required to transmit a plurality of tilevideos to the user or generate a video of an area designated by the userfrom a plurality of tile videos. The former action imposes a burden onthe bandwidth of the user, and the latter action imposes a burden onresources of the server.

Therefore, one of objectives of the present invention to provide anadvantageous technique for efficiently distributing a video of an areaselected by a user.

SUMMARY OF THE INVENTION

An imaging system according to one aspect of the present inventionincludes an imaging device, and a server device that is communicativelyconnected to the imaging device, the imaging device includes at leastone processor or circuit configured to function as an imaging unit thatgenerates a video with a plurality of resolutions, a dividing unit thatperforms a division process of dividing the video generated by theimaging unit into one or a plurality of tile areas to generate a tileimage, and a transmission unit that transmits the video to the serverdevice, and the server device includes at least one processor or circuitconfigured to function as an instruction unit that outputs aninstruction to change a division method for the division process to theimaging device according to a designation frequency of an areadesignated on the video transmitted from the imaging device.

Further features of the present invention will become apparent from thefollowing description of embodiments with reference to the attacheddrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A to 1D are diagrams describing an imaging system according to afirst embodiment.

FIG. 2 is a diagram illustrating an example in which a video generatedby an imaging device is divided into tiles.

FIGS. 3A to 3D are diagrams illustrating examples of videos stored by arecording server.

FIG. 4 is a diagram illustrating a relationship between S_(L) andN_(SL).

FIG. 5 is a diagram illustrating an example of an observation areaaccording to the first embodiment.

FIG. 6 is a block diagram illustrating a configuration example of theimaging system according to the first embodiment.

FIGS. 7A and 7B are flowcharts showing distribution video determinationprocessing of a distribution video determination unit.

FIG. 8 is a flowchart explaining a tile division process according to asecond embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, with reference to the accompanying drawings, favorablemodes of the present invention will be described using Embodiments. Ineach diagram, the same reference signs are applied to the same membersor elements, and duplicate description will be omitted or simplified.

First Embodiment

First, a problem of an imaging system that manages high scalability andbandwidths will be described using FIGS. 1, 2, 3, 4, and 5 .

FIG. 1A to 1D are diagrams describing an imaging system 500 according toa first embodiment. FIG. 1A is a conceptual diagram illustrating anexample of a configuration of the imaging system 500 that manages highscalability and bandwidths. FIGS. 1B to 1C each are diagramsillustrating an example of a tile video group transmitted by an imagingdevice 501 to a recording server 502. FIG. 2 is a diagram illustratingan example in which a video generated by the imaging device 501 isdivided into tiles.

The imaging system 500 is a system including the imaging device 501 andthe recording server 502, which enables an image-capturing scene 101 tobe distributed to a client 104 or another client 105 operated by a user.The imaging device 501 is communicatively connected to the recordingserver 502, and the recording server 502 is communicatively connected tothe clients 104 and 105. Here, although the clients are disposed outsidethe imaging system 500, they may be incorporated into the imaging system500. In addition, although the number of clients is limited to two forconvenience in the description, it may be one or more.

The imaging device 501 captures a image-capturing scene 101 with aplurality of resolutions, generates a video (image) with a plurality ofresolutions, and transmits it to the recording server 502. The imagegenerated by the imaging device 501 is turned into, for example, animage 106 with a low resolution, an image 107 with an intermediateresolution that is higher than that of the image 106, and an image 108with a resolution that is higher than that of the image 107.

Here, it is assumed that, as a resolution for imaging becomes higher, alayer number increases by setting the image 106 with the lowestresolution to a layer 1 image and the image 107 with the intermediateresolution to a layer 2 image. For example, the imaging device 501 maydivide an image into one tile (a tile area) as the layer 1 image. Thelayer 1 image can be said as being non-divided because it has one tileafter division.

In addition, the layer 2 image is divided into 2×2 tiles, a layer 3image is divided into 4×4 tiles, and each of the tile images iscompressed using a compression method, for example, H.264, etc., andtransmitted to the recording server 502. Here, if the maximum number oflayers is set to 3 and the number of division tiles in an i-th layer isset to ni, n1=1, n2=4, and n3=16 as illustrated. Here, assuming that alayer number of a layer to which a tile image generated by the imagingdevice 501 belongs is i and a value indicating a position of the tile isj, the tile image is expressed as Aij.

Here, a position of a tile is numbered in the raster scan order, forexample, in which the upper-left tile serving as a starting point isgiven “1” as illustrated in FIG. 2 and the tiles on the right side aregiven the following numbers. Then, the recording server 502 receives andrecords Σni videos per one frame time. In addition, in the presentspecification, a layer with a higher layer number than a certain layer Kis referred to as a lower layer with respect to the certain layer K, anda layer with the highest layer number is referred to as the lowestlayer.

Meanwhile, the client 104 or the client 105, or both of them transmit arequest for acquiring video data to the recording server 502. Uponreceiving a video acquisition request from the client 104, for example,the recording server 502 first transmits an entire image-capturing scenevideo of the layer 1 (the image 106) to the client.

FIGS. 3A to 3D are diagrams illustrating an example of videos stored bythe recording server 502. FIG. 3A is a diagram illustrating examples ofvideos acquired by the recording server 502 from the imaging device 501.FIGS. 3B and 3C are diagrams illustrating examples of videos displayedon client screens. In the first embodiment, the imaging device 501divides a captured video into predetermined tile areas as illustrated inFIG. 3A, as an example.

Here, a video 300 of FIG. 3B is assumed to be a layer 1 video displayedon a display device (client screen) of the client 104. It is assumedthat a user operates an input device (a mouse, a touch panel, etc.)provided in the client, which is not illustrated, to set (designate) anarea 301 included in the video (here, the video 300) displayed on theclient screen as an area to which the user wants to pay attention(referred to as an “observation area” below). Further, here, anobservation area refers to an area designated by a user, and can also bereferred to as a designated area.

At this time, the client 104 transmits a video acquisition commandhaving information for identifying the observation area to the recordingserver 502. Receiving the video acquisition command, the recordingserver 502 transmits a video of a layer 2 tile A21 corresponding to theobservation area (here, the area 301) to the client 104.

As a result, the video displayed on the client 104 changes into a video303. In addition, when the user watching the video 300 sets an area 302as an observation area, the recording server 502 transmits a video of atile A32 that belongs to a layer 3 to the client 104. As a result, thevideo displayed on the client 104 changes into a video 304.

Further, the expression “a video of a tile A21” to be transmitted to theclient will be referred to simply as a “tile image A21” below. Here,when the user sets an area 305 as an observation area, for example, therecording server 502 transmits tile images A21, A22, A23, and A24 of thelayer 2. Then, the client 104 needs to decode the tile images A21, A22,A23, and A24 to cut out and display the area portion of the area 305.

This case indicates that the bandwidth of the recording server 502 andthe client 104 is four times that of area designation sufficient fortransmission of one tile image. In a case of many image-capturing scenesfor which a user sets an area that requires multiple tile images fordisplay (e.g., the area 305) as an observation area as described above,the efficiency deteriorates.

Thus, the first embodiment presents a system in which a tile divisionmethod is switched according to a set frequency at which a user sets(designates) an observation area (designation frequency) and the area isdistributed to have a resolution that the user desires with highefficiency.

Here, in a general case where an image is divided into M×M tiles and avideo is created by combining tiles, a video according to the followingformula 1-1 is conceivable.

$\begin{matrix}{N_{M} = {{\sum\limits_{k = 0}^{M - 1}{\left( {M - k} \right)\left( {M - k} \right)}} = {\frac{1}{6}{M\left( {{2M^{2}} + {3M} + 1} \right)}}}} & {{Formula}1 - 1}\end{matrix}$

Here, N_(M) indicates the number of videos obtained by combining tileswhen an image is divided into M×M tiles, that is, the number of videosgenerated by combining tiles. However, it is assumed that spatiallyconsecutive tiles are used in the combination of tiles, and an aspectratio of a video generated by combining tiles is equal to the aspectratio of the tiles before being combined.

Meanwhile, a case will be considered where the number of division tilesdoubles each time the number of layers increases by one. The number ofvideos U_(L) for a layer L may be expressed as the following formula1-2.

U _(L)=2^(2(l−1))  Formula 1-2

Here, l indicates a value of the layer. However, the number of videosfor the layer l is set to one. Then, a total number of videos SL may beexpressed as the following formula 1-3.

$\begin{matrix}{S_{L} = {{\sum\limits_{i = 1}^{L}2^{2{({i - 1})}}} = {\frac{1}{3}\left( {2^{2L} - 1} \right)}}} & {{Formula}1 - 3}\end{matrix}$

Here, M=S_(L)=2^((L−1)) is satisfied, and a difference between the totalnumber of videos SL when the number of division tiles doubles each timethe number of layers increases by one and a total number of videosN_(SL) when a video is created by combining tiles exponentially expandsas a layer number L increases, as shown in FIG. 4 . Further, FIG. 4 is agraph showing a relationship between S_(L) and N_(SL). In this graph,the vertical axis represents a total number of videos, and thehorizontal axis represents a layer number.

Here, if N_(SL) tile images are prepared in the recording server 502, aposition of a tile video can be designated and transmitted to set theresolution that the user wants to a resolution corresponding to thenumber of images in the lowest layer, and thus distribution can beperformed with good efficiency. However, because the bandwidth of theimaging device 501 and the recording server 502 is limited and theimaging device 501 has a limited capability, the number of videos to betransmitted to the recording server is also limited in reality.

Thus, a method in which the total number of videos is fixed to S_(L), aconfiguration of tile images of each layer being changed according to asetting frequency of an area set as an observation area by a user and aposition thereof is introduced. Here, although the system in which thenumber of videos increases by 2×2 times each time the number of layersincreases by one has been introduced for simplification, it is merely asimple configuration, and any value may be adopted as long as theincrease rate is fixed. That is, in the first embodiment, the number ofvideos becomes greater as the resolution becomes higher. In other words,a ratio between the total numbers of tile images in the layers iscorrelated to a ratio between resolutions of the layers.

In a case where an observation area selected by a user has a magnitude(size) of a tile image of a layer K, the recording server 502 expressesthe area as a combination of videos (tiles) of a layer k+1 and storesthe layer. Specifically, this will be described using FIG. 5 . FIG. 5 isa diagram illustrating an example of an observation area according tothe first embodiment.

In a case where an observation area selected by a user is an area 401 asillustrated in FIG. 5 , for example, the observation area has the sizeof a tile image of a layer 2, and is stored as being composed of tileimages A35, A36, A39, and A310 of a layer 3. Here, configurations of allfour tiles of the layer 2 are described as reference tiles of the layer3 that is one layer lower than the layer 2, for example, A21=(A31, A32,A35, A36).

Here, reference tiles are defined by a configuration in which an entireimage-capturing area can be covered with a tile size of a layer K byeach of 2^(2(k−1)) tiles. Reference tile configurations are, forexample, in the layer 3, A21=(A31, A32, A35, A36), A22=(A33, A34, A37,A38), A23=(A39, A310, A313, A314), and A24=(A311, A312, A315, A316). Thereference tile configurations are uniquely determined in all of thelayers.

Here, in a case where a tile image of the layer K is expressed withtiles of the layer K+1, a combination of videos, C_(k), is expressed bythe following formula 1-4.

C _(k)=(2^(k)−1)(2^(k)−1)  Formula 1-4

The recording server 502 arranges areas composed of the C^(k)combinations in the order of areas that are most frequently selected bythe user, and then forms a set of areas Ωk. If the number of selectiontimes of each area is set to n_(i), the total number thereof is set tom_(k), and the total number of selection times of a set Z_(k) including2^(2(k−1)) higher areas is set to M_(k), m_(k) and M_(k) are expressedby the following formulas 1-5 and 1-6, respectively.

$\begin{matrix}{m_{k} = {\sum\limits_{i}^{{({2^{k} - 1})}{({2^{k} - 1})}}n_{i}}} & {{Formula}1 - 5}\end{matrix}$ $\begin{matrix}{M_{k} = {\sum\limits_{i}^{2^{2{({k - 1})}}}n_{i}}} & {{Formula}1 - 6}\end{matrix}$

In addition, a higher probability P_(k) is defined as shown in thefollowing formula 1-7.

$\begin{matrix}{P_{k} = \frac{M_{k}}{m_{k}}} & {{Formula}1 - 7}\end{matrix}$

Furthermore, an area that is not covered by an area having the union ofthe 2^(2(k−1)) higher areas with respect to the entire image-capturingarea is assumed as A_(k). An average number of videos U_(k) of thenumber of videos in the layer lower than the layer k that should betransmitted when an area included in A_(k) is selected is calculated.

In addition, an expected value E_(k) of the number of videos that shouldbe transmitted using reference tiles is calculated. When the conditionfor the following formula 1-8 is satisfied, the recording server 502transmits a command to the imaging device 501 to divide an image intotiles in a reference tile configuration. In other words, the recordingserver 502 transmits a command to the imaging device 501 to performprocessing of dividing an image into predetermined tile areas (referencetiles).

E _(k)<(1−P _(k))U _(k) +P _(k)  Formula 1-8

In the other hand, when the condition is not satisfied, for example, therecording server 502 transmits a command to the imaging device 501 toemploy the set Z_(k) as a tile configuration. In other words, therecording server 502 transmits a command to the imaging device 501 toperform division processing with respect to an observation area.

With this configuration, one tile image may be transmitted in a casewhere there is a bias in a user's selection of an observation area andthe user's selection of the observation area requires transmitting aplurality of tile images in the reference tile configuration.Specifically, for example, a change is made to employ a tile divisionconfiguration in which one tile image including the area 305 illustratedin FIG. 3B is sufficient when the user selects the area 305 many times,and thus highly efficient transmission can be performed.

A configuration example of the imaging system according to the firstembodiment will be introduced with reference to FIG. 6 . FIG. 6 is ablock diagram illustrating a configuration example of the imaging system500 according to the first embodiment. The imaging system 500 includesthe imaging device 501 and the recording server 502. Further, each ofthe blocks illustrated in the drawing is realized by a computer (CPU),which is not illustrated, as a control unit built in each of the imagingdevice 501 and the recording server 502. The constituent elements arerealized by the computer performing computer programs stored in amemory. Further, the clients 104 and 105 are also assumed to include acomputer likewise.

The imaging device 501 includes an imaging unit 503, a dividing unit504, an encoding unit 505, and a transmission unit 506.

The imaging unit 503 captures a plurality of videos with varyingresolutions. In other words, the imaging unit 503 generates videos witha plurality of resolutions. Here, an image with a lowest resolution iscalled a layer 1, and images are called a layer 2, a layer 3, . . . alayer kin an ascending order of resolution. The imaging unit 503 outputsan image of each layer to the dividing unit 504.

The dividing unit 504 divides a video into images of each layeraccording to the number of tiles predetermined for the layer, and apattern (a division method), and outputs the images to the encoding unit505. Here, the division method is set to be selected from among twotypes including a configuration designated by a division control unit507, which will be described below, and a reference tile configurationin which an image is divided into the number of tiles predetermined forthe layer so that the tiles cover the entire area of the image. Here, itis assumed that a predetermined number of tiles n_(i) in a layer i is,for example, 2^(2(i-L)), for simplification.

The encoding unit 505 encodes a group of images input from the dividingunit 504 and outputs the result to the transmission unit 506.

The transmission unit 506 outputs the group of videos input from theencoding unit 505 to the recording server 502.

The recording server 502 includes a video reception unit 511, a videostorage unit 512, a video transmission unit 513, a command receptionunit 510, a distribution video determination unit 509, a calculatingunit 508, and a division control unit 507.

The video reception unit 511 receives a group of videos input from theimaging device 501 and outputs it to the video storage unit 512.

The video storage unit 512 stores the group of videos input from thevideo reception unit 511 in a medium. Here, the medium is assumed to be,for example, a solid state drive (SSD). In addition, a video for whichthere has been a distribution video request from the distribution videodetermination unit 509 is output to the video transmission unit 513.Here, storing a video in a medium is not an essential function, and justoutputting a video for which there has been a distribution video requestalone may be included as a function.

The video transmission unit 513 transmits the video input from the videostorage unit 512 to a requested client.

The command reception unit 510 receives a command for designating anobservation area from the client, and transmits the command to thedistribution video determination unit 509. The command for designatingan observation area may be a command for designation using coordinatesin an entire video of the layer 1, or global coordinates based on thecamera.

The distribution video determination unit 509 determines an area in avideo stored in the video storage unit 512 that is closest to theobservation area designated by the user received by the commandreception unit 510, and transfers the result to the video storage unit512.

Here, a method of selecting a group of tiles that is closest to theobservation area will be described using the flowchart of FIGS. 7A and7B. FIGS. 7A and 7B are flowcharts showing distribution videodetermination processing by the distribution video determination unit509. Further, each operation (step) shown in this flowchart is performedby the computer included in the recording server 502 performing acomputer program stored in the memory.

When receiving an observation area designated by the user from thecommand reception unit 510 from the client, the distribution videodetermination unit 509 expands the observation area into a rectanglecircumscribing the observation area in S601. The rectanglecircumscribing the observation area may be configured by, for example,simply combining a maximum x coordinate value, a minimum x coordinatevalue, a maximum y coordinate value, and a minimum y coordinate value ofthe area. Here, x and y may be parallel to the horizontal and verticalsides of the image. After a rectangle circumscribing the observationarea is determined, the observation area is extended into theabove-described rectangle circumscribing the observation area, and theprocess proceeds to S602.

The size and the central coordinates of the observation area arecalculated in S602, and the process proceeds to S603. Here, thecoordinate system is assumed to be a coordinate system in the layer 1image (the whole image). For simplification, the size may be the longerside length between the length of the x side and the length of the yside and axis information (information of the x side or the y side).

In S603, the side length of the same axis as that of axis information ofone reference tile is compared with the side length of the observationarea in order from the layer 1. This comparison is applied to a lowerlayer until the side length of a reference tile becomes shorter than theside length of the observation area. The layer one above the layer inwhich the side length of the reference tile is shorter than the sidelength of the observation area is determined as a transmission tilelayer, and the process proceeds to S604. Further, when the side lengthof the observation area is shorter than the side length of the referencetile in the lowest layer, the lowest layer is set as a transmission tilelayer.

The layer one below the transmission tile layer is set as a compositiontile layer in S604, and the process proceeds to S605. Further, in thecase where the lowest layer is a transmission tile layer, thecomposition tile layer is set as the lowest layer.

A group of composition tiles including a plurality of tiles thatcompletely cover the observation area is designated in a reference tilecomposition of the composition tile layer in S605, and the processproceeds to S606.

In S606, it is checked whether there is a tile including all of thegroup of composition tiles in the transmission tile layer. If there issuch a tile, the process proceeds to S607. If there is no such tile, theprocess proceeds to S608. One tile including all of the group ofcomposition tiles is selected as a tile to be transmitted in S607, andthen the process ends.

In S608, it is checked whether there is a tile including a group ofcomposition tiles in the transmission tile layer. If there is such atile, the process proceeds to S609, and if not, the process proceeds toS613.

In S609, it is checked whether the tile includes all of the group ofcomposition tiles in combination of a tile in the transmission tilelayer including the group of composition tiles. If the tile can includeall of the group of composition tiles, the process proceeds to S610, andif the tile includes only some of all of the group of composition tiles,the process proceeds to S611.

Among combinations of tiles that can include all of the group ofcomposition tiles, a combination of the minimum number of tiles isselected as a group of transmission tiles in S610, and then the processends.

In S611, some of the group of transmission tiles are selected.

Specifically, among the tiles of the transmission tile layer includingthe group of composition tiles, a tile of the transmission tile layermost overlapping (having the largest overlapping area) with the group ofcomposition tiles is determined as a transmission tile. The compositiontile included in the determined tile is excluded from the group ofcomposition tiles, and is likewise determined as a transmission tile. Byperforming this method until there is no tile of the transmission tilelayer including the group of composition tiles, some of the transmissiontiles can be determined.

In S612, the composition tile included in the transmission tiles isexcluded from the group of the composition tiles, the group ofcomposition tiles is updated, and the process proceeds to S613.

In S613, it is checked whether the composition tile layer is the lowestlayer. If the composition tile layer is the lowest layer, the processproceeds to S614, and if not, the process proceeds to S615.

The group of composition tiles is selected as a transmission tile inS614, and then the process ends.

On the other hand, the composition tile layer is lowered by one in S615,and then the process proceeds to S616. The group of composition tiles isupdated by setting a group of reference tiles of the composition tilelayer covering the group of composition tiles as a new group ofcomposition tiles in S616, and the process proceeds to S606.

The distribution video determination unit 509 transmits the group oftransmission tiles selected using this method to the video storage unit512 as a group of tiles that is closest to the observation area. Inaddition, the video storage unit 512 outputs a video for which there hasbeen a distribution video request from the distribution videodetermination unit 509 to the video transmission unit 513. The videotransmission unit 513 transmits the video input from the video storageunit 512 to the client.

Returning to FIG. 6 , the calculating unit 508 increments the frequencyof the minimum tile including all of the group of composition tilescomposing the observation area and outputs the result to the divisioncontrol unit 507.

The division control unit 507 determines whether to change a divisionmethod from, for example, 2^(2(k−1)) areas, the number of selections,the total number of selections, and an area that is not covered in alayer k designated with a number k from 1 to the total number of thelowest layer, in a descending order of the number of selections.Specifically, the division control unit 507 determines whether to dividean image in a reference tile composition or to make a tile compositionbased on the designation frequency effective.

Then, an instruction is output to the dividing unit 504 of the imagingdevice 501. In other words, the division control unit 507 functions asan instructing unit that outputs an instruction to change the divisionmethod of the division process to the imaging device 501. Here, forexample, if the formula (1-8) is satisfied, the reference tilecomposition may be set, and if not, a tile composition based on thefrequency may be set.

Further, it is desirable for the division control unit 507 to determinewhether to change the division method of the division process for thevideos having the plurality of resolutions in a descending order of theresolution, excluding the lowest layer, and to output the instruction tochange to the dividing unit 504. With this configuration, optimizationcan be realized in order of videos having higher resolutions, and thushigher efficiency can be achieved.

By switching a tile division method using the above-described method, avideo to be selected as an observation area more frequently can beprovided with higher efficiency without changing the entire bandwidthwhen there is a bias in selection of a user. In addition, an area that auser designates as an observation area more frequently can bedistributed with higher efficiency.

Second Embodiment

As a second embodiment, an imaging device according to a secondembodiment will be described with reference to FIG. 8 . Similar to thefirst embodiment, the second embodiment proposes a system in which atile division method is switched according to a set frequency at which auser sets an area as an observation area and the area is distributedwith a resolution that the user desires.

The second embodiment is characteristic in that information ofrobustness evaluation of a video further photographed to determine atile composition described in the first embodiment with respect toscaling conversion. The second embodiment is different from the firstembodiment in the tile division method selected by the dividing unit 504illustrated in FIG. 6 , is the same as the first embodiment in otherpoints, and thus the part other than a tile division method will beomitted.

A tile division method in a certain K layer that is not the lowest layerwill be described using the flowchart of FIG. 8 . FIG. 8 is a flowchartexplaining a tile division process according to the second embodiment.Further, each operation (step) shown in this flowchart is performed by acomputer included in the imaging device 501 performing a computerprogram stored in the memory. Further, a tile division method for thelowest layer is the same as that of the first embodiment and thus has areference tile composition.

In S801, if a tile of the K layer composed of a group of compositiontiles of the K layer (the same as a group of reference tiles of a K+1layer) is set as Aj and the number of times in which a user selects eachAj is set as nj, a probability Pj at which each Aj is selected iscalculated using the following formula 2-1, and the process proceeds toS802.

$\begin{matrix}{P_{j} = \frac{n_{j}}{\sum_{i = 1}^{{({2^{k} - 1})}{({2^{k} - 1})}}n_{i}}} & {{Formula}2 - 1}\end{matrix}$

In S802, a tile composition candidate R is obtained as follows. A groupof tiles Ap that is an element of a tile set requested from a lowerlayer is obtained, Ap is excluded from the group of composition tilesAj, and the tiles Aj are rearranged in order of tiles having a higherprobability Pj. A group of tiles Ap is added to the head of the finallyrearranged tile Aj, and the tiles are numbered with A′ in an ascendingorder from the head to the tail. Then, tiles from A₁′ toA_(2{circumflex over ( )}(2k-2))′ are tile composition candidates R. Inaddition, initialization is performed with the requested tile set to thek−1 layer as an empty set, and the process proceeds to S803. Here, therequest from the lower layer is determined based on the processingresult of S801 for the k+1 layer.

A coverage rate Ω is calculated using the following formula 2-2 in S803,and the process proceeds to S804.

$\begin{matrix}{\Omega = \frac{\bigcup_{i = 1}^{2^{{2k} - 2}}A_{i}}{\bigcup_{i = 1}^{{({2^{k} - 1})}{({2^{k} - 1})}}A_{i}}} & {{Formula}2 - 2}\end{matrix}$

Here, a union of tiles A and B is, for example, a set of pixels with aunion of A={(x, y)|x1<x<x2, y1<y<y2} and B={(x, y)|x3<x<x4, y3<y<y4}.Here, x1, x2, y1 and y2 are minimum and maximum x coordinate values andminimum and maximum y coordinate values to determine the area of thetile A, respectively, and x3, x4, y3 and y4 are minimum and maximum xcoordinate values and minimum and maximum y coordinate values todetermine the area of the tile B.

In S804, the coverage rate Ω is compared with a certain thresholdΩ_(th). If the coverage rate Ω is lower than the certain thresholdΩ_(t)h, the process proceeds to S806, and if not, the process proceedsto S805. Here, the threshold Ω_(th) may be an arbitrary value, forexample, 0.5.

In S805, a tile composition candidate R may be employed as a compositiontile to be used to divide tiles in the K layer, and the process ends.

A spatial frequency of a tile that is an element of the tile compositioncandidate R composed of tiles from A₁′ toA_(2{circumflex over ( )}(2k−2))′ is calculated in S806, and the processproceeds to S807. Here, the frequency of each tile is set to (ω1, ω2, .. . , ω2^(2(k−1))), and the spatial frequency may be assumed as aspatial frequency in the horizontal direction for simplification, forexample.

A frequency ω that is lower than the frequency of the threshold ω_(th)is obtained to gain a subset B of the tile composition candidates R inS807, and the process proceeds to S808. Further, ω_(th) is a spatialfrequency at which no deterioration in image quality occurs due toscaling conversion. Because a system in which the resolution of one tileis set to be equal in each of the layers is considered forsimplification, the value may be, for example W/4 in this case. Here, Wis the number of pixels of the horizontal side of a tile.

It is evaluated in S808 whether the obtained subset B is an empty set.If the subset B is an empty set (No), the process proceeds to S809, andif not (Yes), the process proceeds to S810.

In S809, a tile composition candidate R may be employed as a compositiontile to be used to divide tiles in the K layer, and the process ends.

A tile of the subset B is obtained as a composition tile, and if areference tile of the K−1 layer does not overlap an element of a K−1requested tile set, the reference tile is added to the K−1 requestedtile set in S810, and the process proceeds to S811.

In S811, the elements of the subset B are removed from the tilecomposition candidates R. Then, NB elements in the same number as thoseof the subset B can be input to the tile composition candidates R of theK layer. Thus, the same NB elements are extracted from the set obtainedby subtracting the tile composition candidates R from the group ofcomposition tiles of the K layer in a descending order of theprobability Pk, they are set as new tile composition candidates R inaddition to the tile composition candidates R, and the process proceedsto S803.

An effective tile composition can be determined using theabove-described method until the coverage rate Ω exceeds the thresholdΩ_(th) or there is no tile set without deterioration even it isrepresented on a higher layer. By switching a tile composition, areference tile, and a tile division method determined using theabove-described method, a video to be selected as an observation area ata higher frequency can be provided with higher efficiency withoutchanging the entire bandwidth when there is a bias in selection of auser.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation toencompass all such modifications and equivalent structures andfunctions. In addition, as a part or the whole of the control accordingto the embodiments, a computer program realizing the function of theembodiments described above may be supplied to the imaging system, orthe like through a network or various storage media. Then, a computer(or a CPU, an MPU, or the like) of the imaging system, or the like maybe configured to read and execute the program. In such a case, theprogram and the storage medium storing the program configure the presentinvention.

This application claims the benefit of Japanese Patent Application No.2021-132446 filed on Aug. 16, 2021, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An imaging system comprising: an imaging device;and a server device that is communicatively connected to the imagingdevice, wherein the imaging device includes at least one processor orcircuit configured to function as: an imaging unit configured togenerate a video with a plurality of resolutions, a dividing unitconfigured to perform a division process of dividing the video generatedby the imaging unit into one or a plurality of tile areas to generate atile image, and a transmission unit configured to transmit the video tothe server device, and the server device includes at least one processoror circuit configured to function as: an instruction unit configured tooutput an instruction to change a division method for the divisionprocess to the imaging device according to a designation frequency of anarea designated on the video transmitted from the imaging device.
 2. Theimaging system according to claim 1, wherein the instruction unitfurther outputs an instruction to change the division method for thedivision process to the imaging device according to a position of thedesignated area on the video.
 3. The imaging system according to claim1, wherein the instruction unit further outputs an instruction toperform the division process on the predetermined tile areas or toperform the division process based on the designated area.
 4. Theimaging system according to claim 1, wherein a ratio between a totalnumber of first tile images generated from a first video having a firstresolution and a total number of second tile images generated from asecond video having a second resolution that is higher than the firstresolution is correlated with a ratio between the first resolution andthe second resolution.
 5. The imaging system according to claim 1,wherein the imaging device further includes at least one processor orcircuit configured to function as: a calculating unit configured tocalculate an expected value of the number of videos to be transmitted tothe server device according to the designated area, and the instructionunit outputs the instruction based on the expected value.
 6. The imagingsystem according to claim 1, wherein the instruction unit excludes avideo having a highest resolution from the videos having a plurality ofresolutions, determines whether to change a division method for thedivision processing in a descending order of resolution, and outputs theinstruction to the imaging device.
 7. The imaging system according toclaim 1, wherein the dividing unit performs the division process basedon information of an evaluated deterioration in image quality caused byscaling conversion.
 8. The imaging system according to claim 7, whereinthe dividing unit acquires a spatial frequency of an area obtained bydivision as the tile area in the evaluation of deterioration in imagequality caused by scaling conversion to perform the division process. 9.The imaging system according to claim 1, wherein the imaging devicefurther includes at least one processor or circuit configured tofunction as: an encoding unit configured to encode a video generated bythe imaging unit and the tile image.
 10. A server device that iscommunicatively connected to an imaging device that generates a videowith a plurality of resolutions, performs a division process to dividethe generated video into one or a plurality of tile areas, and generatesa tile image, the server device comprising: at least one processor orcircuit configured to function as: an instruction unit configured tooutput an instruction to change a division method for the divisionprocess to the imaging device according to a designation frequency of anarea designated on the video transmitted from the imaging device.
 11. Acontrol method for a server device that is communicatively connected toan imaging device that generates a video with a plurality ofresolutions, performs a division process to divide the generated videointo one or a plurality of tile areas, and generates a tile image, thecontrol method comprising: outputting an instruction to change adivision method for the division process to the imaging device accordingto a designation frequency of an area designated on the videotransmitted from the imaging device.
 12. Anon-transitorycomputer-readable storage medium configured to store a computer programto control an imaging device that generates a tile image and a serverdevice communicatively connected to the imaging device, the mediumcomprising instructions for executing following processes: a step ofgenerating a video with a plurality of resolutions and dividing thegenerated video into one or a plurality of tile areas in the imagingdevice; and a step of outputting an instruction to change a divisionmethod for the division process to the imaging device from the serverdevice according to a designation frequency of an area designated on thevideo transmitted from the imaging device.